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Abstract 

Background: The Implementation Research Institute (IRI) provides two years of training in mental health 
implementation science for 10 new fellows each year. The IRI is supported by a National Institute of Mental Health 
(NIMH) R25 grant and the Department of Veterans Affairs (VA). Fellows attend two annual week-long trainings at 
Washington University in St. Louis. Training is provided through a rigorous curriculum, local and national mentoring, 
a 'learning site visit' to a federally funded implementation research project, pilot research, and grant writing. 

Methods: This paper describes the rationale, components, outcomes to date, and participant experiences with IRI. 

Results: IRI outcomes include 31 newly trained implementation researchers, their new grant proposals, 
contributions to other national dissemination and implementation research training, and publications in 
implementation science authored by the Core Faculty and fellows. Former fellows have obtained independent 
research funding in implementation science and are beginning to serve as mentors for more junior investigators. 

Conclusions: Based on the number of implementation research grant proposals and papers produced by fellows 
to date, the IRI is proving successful in preparing new researchers who can inform the process of making evidence- 
based mental healthcare more available through real-world settings of care and who are advancing the field of 
implementation science. 
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Background 

Advancing the science of implementation is a priority in 
mental health, where disease burden is high, the reper- 
toire of effective treatments is growing, but availability 
and receipt of evidence- based care is low. A summary 
of epidemiologic survey data suggests that only 40% to 
50% of people with mental disorders receive any treat- 
ment and of those receiving treatment, a fraction receive 
what could be considered quality' treatment [1-4]. The 
gap between what could be delivered and what people in 
need of mental health services receive— a direct focus of 
implementation science— is vast [5]. While the pace of 
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implementation science development has increased, we 
still see comparatively little investment in health services 
research and even less in the more specific area of im- 
plementation science [6]. 

To accelerate the implementation of evidence-based 
mental healthcare, the field must meet two pressing needs: 
a body of research and theory informing effective imple- 
mentation processes, and a research workforce capable of 
conducting rigorous and relevant implementation studies. 
Over the past decade, the field has experienced what Na- 
tional Institute of Healths (NIH) Dr. David Chambers 
characterized as an explosion' in quality and quantity of 
implementation research [7]. This is demonstrated by 
rigor and ambitiousness of ongoing studies,' more com- 
parative effectiveness studies of active [dissemination and 
implementation] strategies,' and the capacity building of 
multiple research centers and networks' [7]. This paper 
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focuses on the necessary complementary efforts to grow 
the implementation science workforce. It describes the ra- 
tionale, components, outcomes to date, and participant ex- 
periences with the National Institute of Mental Health 
(NIMH) funded Implementation Research Institute (IRI) in 
mental health at Washington University in St. Louis. 

Background, institute aims, and overview 

In 2004, the NIH began convening meetings to advance 
implementation science, but no NIH-funded training 
programs {e.g., T32 programs, other R25 programs) fo- 
cused squarely on this field. Recognizing the dearth of 
investigators prepared to meet the unique conceptual 
and methodological challenges in implementation sci- 
ence, the authors of this paper began working together 
in 2006 to address the need for specialized training, con- 
sider various training approaches, and explore mecha- 
nisms of funding support. 

Several principles and guiding assumptions shaped our 
implementation science training plans. First, because im- 
plementation science was still emerging with rapid and 
continuous knowledge developments for the foreseeable 
future, its training would need to differ from that in other 
fields which could immerse trainees in a mature science. 
We agreed that training would need to be informed by the 
continuous refreshing and infusion of new intellectual 
capital. This would require annual reassessment and pos- 
sible revision of the curriculum, as well as change in roster 
of training faculty. The Core Faculty accepted a second 
guiding principle: that our efforts required us not only to 
stay on top of a rapidly developing field, but also to con- 
tribute to the fields advancement. Thus we agreed that 
the training endeavor itself should help accelerate the de- 
velopment of intellectual capital for the field. Accordingly, 
the Core Faculty agreed to work together to identify gaps 
in implementation science and to develop and deliver new 
presentations at the training institute to be published sub- 
sequently for the broader field. Moreover, our trainees 
would be expected to contribute to implementation sci- 
ences knowledge base. Third, we perceived implementa- 
tion of evidence-based mental healthcare as uniquely 
complex in some ways, given its array of specialty and 
non-specialty platforms and treatment types (acute care, 
pharmacology, psychosocial). Such complexity would de- 
mand a longer than typical training period; thus, we 
planned to provide each cohort with two years of annual 
immersion experiences, enriched by in vivo training 
through site visits to state-of-the-art implementation re- 
search projects specifically selected for fellows' emerging 
research programs. 

Several other assumptions shaped the training plan. Be- 
cause implementation science is inherently multidisciplinary, 
IRI faculty, fellows, and curriculum content would necessar- 
ily draw on several disciplines. Similarly, implementation is a 



multi-level phenomenon, with distinct but interacting and 
nested processes operating at the individual-, team-, 
organizational-, and policy-context levels [8,9]; this should 
shape the curriculum and selection of faculty. We felt that 
the training should have strong national leadership, given 
the existence of only pockets of expertise in implementation 
science around the country. No university-based depart- 
ments were leading the field or had a sizable critical mass' 
of implementation researchers among their faculty and stu- 
dents. Accordingly, we expected that most trainees could 
draw on mentors in their home institutions for grant- 
writing expertise, but probably not expertise in implementa- 
tion science. Finally, the IRI is predicated on a pedagogical 
philosophy that values interpersonal activity and interaction 
as a key to the science-building process [10]. Therefore, our 
training would provide opportunities for interpersonal net- 
working among not only faculty and trainees but also other 
types of mentors and stakeholders in real-world clinical and 
service settings. 

These assumptions led us to design the IRI to address 
two overarching goals: first, to strengthen human capital 
for the field of implementation science through the train- 
ing of a new generation of implementation researchers, 
and second, to advance intellectual capital for the still- 
developing field— through stimulating the production of 
scholarly products such as papers, books, and curriculum 
models. While the first goal is typical of early investigator 
training programs, the second goal was considered neces- 
sary because of the emerging science status of the imple- 
mentation research field. Moreover, the IRI was purposively 
shaped to differ from many other training grants in that the 
purpose is not to teach grant writing to very junior scholars, 
but rather, to help both promising and proven scholars 
make a successful transition to implementation research. 

Guided by these assumptions and the assessment of 
need, the authors of this paper shaped an R25 proposal, 
first submitted to the NIMH in 2007 and eventually funded 
in 2009. The IRI was established with core funding from 
the NIMH R25 grant and supplementary funding from the 
Department of Veterans Affairs (VA). This interdisciplin- 
ary training program provides didactic training, faculty 
mentoring (both local and distance), support and guidance 
for pilot research and grant writing, and additional experi- 
ential learning — all focused on helping participants de- 
velop implementation research projects for competitive 
external funding. Part of the experiential learning involves 
observation of agency- and research- implementation ef- 
forts in mental health settings. Fellows are PhD and/or 
MD investigators from a variety of health-related fields, 
with demonstrated experience and enthusiasm in the 
study of mental healthcare, who wish to conduct ground- 
breaking research in the area of implementation science. 
Next, we describe the IRI components, outcomes to date, 
and participant experiences with the Institute. 
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Methods 

Data shown below was approved by the Washington Uni- 
versity Human Research Protection Office (IRB ID #: 
201204044). Informed consent was exempt for this study. 

Guiding definitions 

The Institute is guided by definitions used by the NIH. 
Specifically, we understand implementation as 'the use of 
strategies to adopt and integrate evidence-based health in- 
terventions and change practice patterns within specific 
settings' [11] and implementation research as scientific in- 
vestigations that support movement of evidence-based, ef- 
fective healthcare from the clinical knowledge base into 
routine use [12]. While the IRI focus is on implementation 
research, content addresses dissemination research as well, 
defined by the NIH as 'the study about how, when, by 
whom and under what circumstances research evidence 
spreads throughout the agencies' [13]. 

IRI participants 
Leadership 

The Institute is led by a Director, Associate Director, 
and a Coordinator. Enola Proctor, PI of the NIMH R25 
grant, is IRI Director. The Associate Director is John 
Landsverk, also Director of the Child and Adolescent 
Services Research Center in San Diego. Drs. Proctor and 
Landsverk bring to their leadership roles several decades 
of experience directing research centers, implementation 
research studies, providing academic administration, and 
leading formal training programs (including NIH T32 
programs). A PhD level Coordinator, Ana Baumann, is 
based at Washington University in St. Louis, as is Dr. 
Proctor. Together they lead Institute program adminis- 
tration, including fellow recruitment; planning and or- 
ganizing training events, and evaluation of the program. 

Core faculty 

In addition to Proctor and Landsverk, four nationally 
renowned implementation researchers were recruited to 
serve as members of the IRI Core Faculty for the five years 
of the funding period: Gregory Aarons, University of 
California, San Diego (Psychiatry); Ross Brownson, 
Washington University in St. Louis (Public Health); Charles 
Glisson, University of Tennessee (Social Work); and Brian 
Mittman, Department of Veterans Affairs (Management 
Theory). Each has an established, externally funded re- 
search program relevant to the implementation of 
evidence-based practice (EBP) and a strong record of train- 
ing postdoctoral fellows and junior faculty for independent 
research careers. 

Members of the Core Faculty work as a group to chart 
the long-range direction of the IRI, assess adequacy and 
gaps in training aims, and identify opportunities for col- 
laboration and partnership. They constitute the fellow 



selection committee and each serves as primary mentor 
for specific fellows. The Core Faculty accomplish these func- 
tions through virtual and in-person meetings throughout 
the year. Each leads one or more sessions at the annual In- 
stitute. Program staff from the NIMH and VA are invited to 
each Institute, participating as Expert Faculty. The NIMH 
Associate Director for Dissemination and Implementation 
Research, Dr. David Chambers, has made presentations each 
year. Additionally, he has contributed actively between Insti- 
tutes to requests from Core Faculty to help identify potential 
expert faculty and sites for learning visits. 

To inform curriculum revisions, the Core Faculty hold 
periodic telephone and in-person meetings, including one 
meeting during each Institute focused specifically on 
assessing the curriculum and identifying new areas of em- 
phasis in future years. Curriculum reassessment is also in- 
formed by annual evaluations performed by both faculty 
and fellows. 

Expert faculty 

For each annual Institute, an additional eight to ten indi- 
viduals are invited to serve as Expert Faculty, chosen by 
IRI Directors and Core Faculty for their ability to bring ex- 
pertise in targeted curricular areas at the Institute. The 
composition of Expert Faculty changes from year to year, 
guided by the Core Faculty's on-going assessments of cur- 
ricular needs and progress in the field. Members of the 
Expert Faculty illustrate their ongoing mental health im- 
plementation research in a range of private, VA, and other 
public community and acute care settings. In the first 
three years. Expert Faculty have represented the disci- 
plines of clinical psychology, organizational psychology, 
sociology, communication theory, medical anthropology, 
psychiatry, epidemiology, medicine, social work, public 
health, and electrical engineering (see Table 1 for details). 
Expert Faculty typically spend two to three days at the In- 
stitute, networking and advising fellows in addition to lec- 
turing on specific topics. 

Fellows 

Ten fellows are selected each year, eight supported by the 
NIMH funding and two by VA funding. Information about 
the IRI and application process is widely disseminated 
through relevant email distribution lists, listservs, word-of 
-mouth, and conference announcements; the application 
period is open for four months, typically from the end of 
October to February. The application requires a letter de- 
scribing the applicants interest in implementation re- 
search in the context of their career goals and long range 
research agenda, an initial concept paper in the area of 
mental health implementation science that the applicant 
expects to advance through the IRI toward submission to 
a federal funding agency, a description of resources and 
supports in his or her home institution for grant-writing 
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Table 1 Disciplines from core and expert faculty, years 



2010, 2011 and 2012 

Discipline Number of people 

Medicine 6 

Psycliology'^ 6 

Social Worl< 4 

Psycliiatry 3 

Sociology 2 

Medical Anthropology 2 

Public Health 2 

Epidemiology^ 3 

Organizational Behavior and Management 1 

Communication Therapy 1 

Electrical Engineering, Mathematics 1 

Administration 1 



Note: ''five faculty from Psychology, and one faculty from Industrial/ 
Organizational field; ^one faculty from Epidemiology, one faculty from 
Epidemiology and Public Health, and one from Nutrition Intervention and 
Policy; Epidemiology. 

fundamentals and for authoring scholarly publications, 
and a letter from a local mentor in their home institution. 
Selection criteria include: prior or concurrent experience 
relevant to IR such as intervention development and/or 
testing, mental health services research, or study of 
organizational factors in mental health service delivery; ex- 
perience writing an NIH, VA, or other federal grant; a 
strong local mentor in the applicant s home institution who 
is supportive of the fellows grant writing and scholarly pub- 
lication (the local mentor need not be an expert in IR, but 
must have a strong record of NIH or VA funding); and ac- 
cess to a clinic/service setting willing to serve as a pilot site 
for the fellows implementation research. Applications are 
rated using the NIH grant review criteria where Core Fac- 
ulty review applicants current research for evidence of a 
path' toward dissemination and implementation research, 
prior accomplishments (training agenda vitae, previous 
funding, letters of recommendation/support), and potential 
for a productive implementation research career. Prefer- 
ence is given to applicants who have NIH or VA grant- 
writing experience, have secured funding {e.g., F32, K, R03, 
R34, R21, ROl or similar mechanisms in the VA) in fields 
related to implementation research, and demonstrate po- 
tential to succeed in the Institute. Core Faculty independ- 
ently score each application and final selections are made 
during a conference call, wherein they discuss applications, 
finalize selections, and make preliminary assignments of 
primary mentors. Applicants and selected fellows are noti- 
fied within a few days of the decision, and each selected fel- 
low is asked for a commitment to attend two consecutive 
Institutes. IRI completion requires full participation in the 
two-year program. 



Selected fellows comprise a diverse group of partici- 
pants from various disciplines {i.e., psychology, social 
work, pediatrics, psychiatry, anthropology; epidemiology; 
see Table 2 for more details), institutions, and geography to 
build a national network of implementation researchers. 
All IRI fellows have entered the program with at least one 
externally-funded grant, primarily to eliminate the need to 
address basic grant writing and enable it to focus on imple- 
mentation science. While all fellows enter the IRI at early 
stages of implementation science knowledge, they vary in 
terms of career stage and prior accomplishments. Fellows 
range from one year out of their terminal degree programs 
(PhD, post-doc, or M.D.) to Full Professor. Several entered 
with more limited publication records (minimum number 
of publications for an entering fellow to date is 13), but 
more experienced fellows had more than 50 publications. 
Fellows similarly vary in prior research grant experience. 
Many enter with one funded grant (and for one fellow, this 
was an F31 dissertation grant), while several have received 
R03s, R34s, and career development awards. However, 
none of these grants were in implementation science. This 
marked variation in background and experience poses 
challenges for the development and delivery of the training 
program, as detailed below. 

IRI components 
Annual institute 

The IRI s key training component is an annual five-day In- 
stitute each summer at Washington University in St. Louis. 
The Faculty considered core content areas required in im- 
plementation science as they wrote the R25 grant proposal 
and published a guiding heuristic framework for implemen- 
tation research in mental health [8]. This framework served 
as the model for our identification of core areas, which have 
remained constant over the first three Institutes. The core 
areas include: theoretical bases of implementation science, 
and ability to select, critique, and use an established 
conceptual model or framework to guide a research study; 
the distinction between, and the evidence for, two 
practice-change technologies: clinical/behavior change in- 
terventions, and implementation strategies; contextual 

Table 2 Disciplines from 2010, 2011, and 2012 IRI fellows 



Discipline Number of people 



Psycliology^ 


19 


Social worl< 


4 


Medicine 


3 


Psycliiatry 


2 


Epidemiology 


1 


Anthropology 


2 



Note: fifteen fellows are from Clinical Psychology, one from Social 
Psychology, one from Counseling Psychology, one from Health Psychology, 
one from Applied Experimental Psychology. 
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factors influencing implementation, such as organizational 
context and leadership, and the ability to understand 
whether a given study will observe, control, or manipu- 
late those factors; the multi-level nature of practice or 
service system change, and ability to select appropriate 
research methodologies, including randomized control 
designs and alternatives to group randomization, 
measurement of implementation strategies and out- 
comes, multilevel modeling, mixed methods including 
community based participatory approaches, and alter- 
native sources of data for implementation research; 
understanding key partners in implementation re- 
search and skills required to engage them in programs 
of implementation research; and mounting IR pro- 
grams of research; and understanding of and ability to 
address the specific ethical issues in IR and capacity 
for responsible conduct of science. 

Specific topics within these key areas are adjusted from 
year to year according to advances in the field, the unique 
interests and experience level within each IRI cohort, and 
the presence of new and returning fellows each summer 
after the first year. For example, in year one. Expert Fac- 
ulty member James Bearing gave an overview of Rogers' 
Diffusion of Innovation Theory. The ensuing discussion 
raised the challenge of understanding, critiquing, and 
selecting a particular theory or conceptual model given 
the tendency at that time for implementation researchers 
to develop unique models for each grant application. Ac- 
cordingly, for the year two Institute, David Chambers and 
Ross Brownson developed and presented a session on 
frameworks used in NIH funded D&I research. From this 
session, they collaborated to write and publish a paper 
overviewing and comparing 62 models and theories— a 
paper that became required reading thereafter [14]. 

Given the multidisciplinary and rapidly evolving na- 
ture of implementation science, the Core Faculty spends 
nearly a full year planning each year s curriculum. The 
Core Faculty keep abreast of developments in implemen- 
tation science for mental health, consult with NIMH 
program officers, and nominate and invite new Expert 
Faculty based on their own learning and experiences 
throughout the year. To accelerate fellows' appreciation 
of the realities of conducting implementation research, 
the curriculum draws heavily on current examples of 
funded research in the field. Each year s curriculum is 
also shaped by fellows' needs, interests, and characteris- 
tics. The format and combinations for didactic and ex- 
periential teaching methods is informed by evaluations 
of the previous Institutes. Each Institute employs a var- 
iety of learning formats, including didactic presentations 
by faculty, presentations by fellows, peer critique and 
feedback, informal networking, small group interactions, 
and brief consultation with the faculty' sessions ar- 
ranged through a sign-up process. The week-long 



Institute in St. Louis is an 'immersion' experience pro- 
viding opportunity for strategic networking. For ex- 
ample, each Institute provides an afternoon for fellows 
to consult with Core and Expert Faculty on a sign-up 
basis, and Faculty are available during evening hours for 
further consultation on the grant proposals fellows de- 
velop during the Institute. 

Since the second year, each Institute comprises a co- 
hort of returning second year' fellows along with a new 
cohort of 'first year' fellows. We assume that first year 
fellows, albeit with research experience and some grant 
writing success, may be relative 'novices' in IR, while 
second year fellows come to their second Institute at a 
more advanced level. Accordingly, the curriculum is 
structured to provide optimal redundancy but minimal 
repetition of basics for second year fellows [15,16]. For 
all topics that are repeated in the second year (such as 
research designs for implementation studies), the par- 
ticular focus, examples, readings, exercises, and expert 
faculty differ significantly to convey emerging issues and 
recent advances in implementation science. This ap- 
proach increases depth and breadth of perspective across 
the two years that each fellow experiences. Second year 
fellows are engaged in some teaching [17], and actively 
report on their interim learning. Over the five intensive 
days, each fellow is expected to advance a concept paper 
(first year fellows) or grant proposal (second year fel- 
lows), developing aims and research questions for men- 
tal health implementation science, shaping conceptual 
models, and planning details for the study methodology. 
During the Institute, second year fellows receive exten- 
sive and critical reviews of their proposal by reviewers 
experienced in the NIH review process. These reviews 
serve to normalize the process of critical feedback, 
sharpen ability to read critically, facilitate progress in 
writing proposals, enable students to send draft pro- 
posals to NIMH staff for feedback, and ultimately sup- 
port success with external funding. By the week's end, all 
fellows are expected to have developed an action plan 
for advancing their IR grant proposal. Fellows' presenta- 
tions and revised plans for grant proposal development 
serve as the reflection, within the Institute, of their un- 
derstanding of core content areas and their preparedness 
for implementation research. 

Pre-institute preparation 

Approximately three weeks before the Institute, each fel- 
low and faculty receives an electronic course packet 
containing IRI objectives, a biographical sketch of each 
participant, readings that correspond to each session, 
and the training agenda. Core faculty receive first year 
fellows' IR project concept paper; and second year fel- 
lows and faculty receive the fellows' grant proposals for 
review. Second year fellows also provide their report of 
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learning site visits with Core Faculty (see below). The 
content of these reports are embedded in the curriculum 
of the Institute as current examples of IR studies. The 
packet launches distance learning and reduces the bur- 
den for both faculty and trainees during the Institute. 

Mentoring 

Fellows are matched with an Institute mentor (one of the 
six Core faculty members) with whom they meet by 
phone- or video-conference call monthly to bi-monthly 
during the two year training. Mentoring focuses on the 
progress of the pilot study, the development of the fellows 
IR research proposal, and feedback from NIH program 
staff around the developing grant proposal. To facilitate 
continued application of content from the Institute, all fel- 
lows and faculty have access to the videos of the lectures 
from all previous IRI years via a secure website. 

IR site visits 

Good research training requires the opportunity to learn 
from projects that are scientifically innovative, important, 
and challenging [18,19]. We view active, still-in-the-field 
projects that have passed stringent peer review as an excel- 
lent source of such learning. Therefore we provide each first 



year fellow with travel funds to support a two to three day 
visit to an active implementation research site supported by 
highly competitive federal funding. Working closely with 
NIH and VA program officers and informed by NIH Re- 
porter searches. Proctor, Landsverk and Baumann identify 
sites of state-of-the-art IR that reflect good science and part- 
nership models. See Table 3 for examples of learning sites 
that fellows have visited. 

Prior to their visits, we provide webinar training to all 
fellows to optimize their ability to learn from these visits. 
Each fellow completes two assignments from these visits: 
a site visit report on topics deriving from the training 
agenda, including key concepts (is the project conceived 
as an implementation study, an adaptation of an EBP, dis- 
semination, or test of practice effectiveness?), the study's 
conceptual model (explicit or implicit), the EBP(s) being 
implemented, the implementation processes or strategies 
in place, implementation barriers, the implementation 
agency's characteristics, clinical and implementation out- 
comes measured [8], the type and elements of the design 
employed [20], and the stakeholders and partnerships ex- 
emplified in the project; and a scientific biography of the 
IR study's PI, including educational history and training, 
transition points evident in his/her research career. 



Table 3 IRI learning sites 



PI 


Title 


Funding source 


Location 

(Organization State) 


Sonya Leathers 


Dissemination of effective MH services in foster care 


NIMH; ROl 


Chicago, IL 


Philip Kendall 


Disseminating evidence-based practice to the schools: CBTfor 
child anxiety 


NIMH; ROl 


Philadelphia, PA 


Madhukar 
Trivedi 


Using information technology to provide measurement based 
care for chronic illness 


AHRQ; R18 


Dallas, TX 


Joann Kirchner 


Blended facilitation to enhance PCMH program implementation 


VA 


Little Rock, AR 


John Weisz 


Youth mental health network 


The John D. and Catherine T. 
MacArthur Foundation 


Honolulu, Hawaii, 
Boston, Mass. 


Jeffrey Epstein 


Evaluation of an intervention for improving community-based 
pediatric ADHD care 


NIMH; ROl 


Cincinnati, OH 


Geoffrey 
Curran 


Training SUD treatment counselors CBT for depression 


VA 


Little Rock, AR 


Gregory Aarons 


Organizational issues in implementing EBP in child welfare and 
mental health 


NIMH 


California, San Diego 


Gregory Aarons 


Mixed-methods study of EBP sustainment in Statewide service 
system 


NIMH; ROl 


California, San Diego 


Mark Chaffin 


SafeCare home-based service models 


NIMH; ROl 


Oklahoma City, OK 


Patricia 
Chamberlain 


Experiment in implementing multidimensional treatment foster 
care 


NIMH/NIDA ROl 


California, Ohio 


John Fortney 


Partnership for implementation of evidence-based practices in 
rural primary care 


NIMH; ROl 


Little Rock, AR 


Kimberly 
Hoagwood 


Implementation of feedback system to improve EBTs for 
children in mental health 


AHRQ; R18 


New York 


Richard Owen 


Monitoring and Management of Metabolic effects of 
Antipsychotics 


VA 


Little Rock, AR 


Alex Young 


Implementing effective, collaborative care for Schizophrenia 


VA 


Los Angeles 
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funding history and key publications, and team compos- 
ition. When they return for their second year Institute, 
these senior' fellows share with other fellows the concep- 
tual and methodological developments apparent in the 
implementation studies they have visited. Second year fel- 
lows' feedback guide revisions and updates to the IR site 
visit assignments. The site visits also enable Core Faculty 
to identify an emerging network of IR investigators to in- 
clude in the IRI as possible Expert Faculty. 

Pilot research in real-world implementation 

Each fellow receives funding to conduct pilot work in a 
clinic/service setting that is implementing EBPs and is 
willing to serve as a pilot study site. IRI Faculty help nego- 
tiate fellows' agency experiences. The formal curriculum 
and faculty mentoring guide trainees through IRB issues 
and processes associated with IR. This component of the 
IRI initiates fellows to the partnerships with real world 
sites required of all implementation researchers, and the 
relationships that are fostered are expected to develop into 
longer-term, grant-related partnerships. 

Face-to-face meeting for networking at an IR conference 

First and second year fellows receive support to attend 
an implementation science conference, such as the NIH 
Conference on the Science of Dissemination and Imple- 
mentation in Health Implementation or the Seattle 



Implementation Research Conference (SIRC). All Core 
Faculty are encouraged to attend the conference, which 
provides a platform for formal presentations of fellows 
and faculty's work, as well as informal gatherings. Core 
Faculty and fellows attend an informal networking dinner, 
where they network and recognize graduating senior' fel- 
lows for their accomplishments and contributions. 

In summary, conceptualizing implementation science 
as multi- and trans- disciplinary, the goal of the IRI is to 
ensure that each fellow benefits from a confluence of six 
training environments during their two years of training, 
as shown on Figure 1: the IRI host site, the Washington 
University Center for Mental Health Services Research 
(CMHSR)— the 'intellectual and training hub' and the 
coordinating center; the fellow's home environment, 
which will commit to mentoring and supporting grant 
development; national sites of NIH-fimded implementa- 
tion research hosting fellows' learning site visits, includ- 
ing but not limited to the research settings of some IRI 
Faculty, generating new advances in implementation sci- 
ence; local clinics/agencies settings as pilot sites for fel- 
lows; and interim face-to-face and virtual collaborative 
interactions. 

IRI evaluations 

IRI processes and outcomes are evaluated for fellows, 
mentors, and the program. After each Institute, all fellows 





Figure 1 IRI training environments. 
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and faculty are asked to evaluate the training and provide 
feedback on the logistics of the institute {e.g., organization 
of the schedule, clarity of instructions), on the formal train- 
ing (/.a, on each of the sessions), on the informal training 
(e.g., networking, caliber of the fellows), and on mentoring 
(e.g., have fellows received feedback on their projects? Have 
faculty provided feedback on fellows' projects?). 



implementation researchers. Accordingly, we posit out- 
comes at three levels: participating fellows, participating 
faculty, and for the fields of mental health and implemen- 
tation research. This paper overviews outcomes to date in 
each category. At the end of year five, we will prepare a 
more detailed analysis and findings regarding these three 
levels of outcomes for a subsequent paper for publication. 



Results 

Our evaluations show that fellows and faculty have been 
very satisfied with the program and would recommend 
IRI to colleagues. Fellows and faculty have rated the cali- 
ber of their counterparts as excellent on all three initial 
years (See Table 4 for more details). Participant feedback 
regarding the program has shaped the next year s curricu- 
lum. For example, because fellows and faculty requested 
more time to network and 'digest' the contents learned for 
their own projects, we reduced the number of didactic 
classes per day, and increased formal and informal oppor- 
tunities to network and discuss research projects with fac- 
ulty and peers. 

The overall goals of the IRI are to concurrently advance 
the science of implementation, especially in the area of 
mental health, and develop the supply and capacity of 

Table 4 Average rating from 2010, 2011, and 2012 IRI 
evaluations by fellows 





Average 


SD 


Range 


Logistics 










Timeliness of acceptance decisions 


4.67 


0.47 


3- 


-5 


Adequacy of information 1 received prior to tine 
IRI to prepare me for the IRI 


4.49 


0.65 


3- 


-5 


Organization of the IRI schedule 


4.29 


0.65 


3- 


-5 


Informal Training 










Professional caliber of other fellows 


4.96 


0.20 


4- 


-5 


Collaboration with fellows 


4.67 


0.52 


4- 


-5 


Interactions with non-core faculty 


4.65 


0.60 


3- 


-5 


Mentoring 










Likelihood of pursuing further professional 
relationships with faculty and fellows 


4.88 


0.39 


3- 


-5 


Interactions with core faculty 


4.76 


0.52 


3- 


-5 


Helpfulness of feedback from faculty on pilot 
and NIH proposal 


4.66 


0.60 


3- 


-5 


Helpfulness of feedback from faculty on my 
ideas 


4.63 


0.60 


3- 


-5 


Helpfulness of peer support from other fellows 
on ideas 


4.47 


0.68 


3- 


-5 


Assistance received with study design 


4.30 


0.72 


3- 


-5 


Overall 










Likelihood of recommending IRI to colleagues 


4.92 


0.35 


3- 


-5 


Satisfaction with IRI program 


4.84 


0.42 


4- 


-5 



Outcomes for participating fellows 

Over its first three years, the IRI has recruited three co- 
horts for a total of 31 fellows (in the first year, IRI had 11 
slots, followed by 10 slots for years two and three). These 
fellows have been selected from a total pool of 86 appli- 
cants from a wide range of academic settings and disci- 
plines as shown in Table 2. Fellows from the first three 
cohorts (n = 31) completing their two-year training pe- 
riods (2009-2011, 2010-2012, and 2012-2013) have sub- 
mitted 74 proposals for scientific projects and received 52 
(70%) funded awards. The funding mechanisms varied: 
21% were NIH ROl grants, 21% VA projects, 11% NIH 
R34 grants, 8% NIH K grants, and the remaining 34% are 
grants from other mechanisms, such as CDC, AHRQ, and 
Fogarty (Please see Table 5). The new research supported 
by the IRI training will generate new findings in the emer- 
ging field of implementation science with promise to ac- 
celerate the availability and quality of evidence -based 
mental healthcare. 

Within the first year (for the first two cohorts of fellows) 
and six months (third cohort) after completing the IRI, 
the three cohorts submitted and/or published a total of 
208 publications (mean of 7.64 publications per fellow) re- 
lated to the field of implementation science. They also 
made national presentations in the VA QUERI Cyber Sem- 
inar series and at numerous conferences, and delivered 



Table 5 Outcomes of 2010, 2011, and 2012 cohort of IRI 
fellows 



Outcomes 

Grants 



Numbers 

Number of grants 
submitted: 74 



Number of grants 
awarded: 52 (70%) 



Additional information 

Percentage of type of 
grant awarded: 

21% NIH ROl 

21% VA projects 

11% NIH R34 

8% NIH K 

39% other sources or 
mechanisms {e.g., NIMH 
R21,SAMSHA, AHRQ, 
CDC, Forgaty, Melinda 
and Bill Gates Foundation) 



Publications 



Total number of 
publications: 208 



Note: 1 = very poor; 2 = poor; 3 = fair; 4 = good; 5 = very good. 



Other 63 presentations at 

accomplishments conferences and other 
accomplishments 
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seminars and lectures in their local institutions. These 
contributions reflect IRI fellows' preparedness to address 
the conceptual, methodological, and practical challenges of 
studying the implementation of evidence-based mental 
health interventions. While the publication and grant 
award rates for IRI fellows are clearly impressive, we urge 
caution in the attribution of accomplishments to the IRI. 
An explicit aim for year five of the IRI award is to conduct 
detailed evaluations by cohort in an effort to identif)^ fellow 
and training factors associated with various types and tra- 
jectories of outcomes. Those analyses will cover a longer 
period of time, which will capture fellows' outcomes be- 
yond their first year. We fully expect their contributions to 
grow over time. 

Outcomes for participating faculty 

In its first three years, the IRI has recruited 27 Expert 
Faculty from a wide range of disciplines (see Table 1) to 
join the six Core Faculty in providing Institute training. 
Each faculty member leads one or more sessions in the 
week-long Institute and works to establish relationships 
with each other and with feflows. For example, although 
we ask Expert Faculty members to spend only two days 
at the Institute (whereas Core Faculty members attend 
all five days), each year several experts have extended 
their stay and actively participated in discussion, net- 
working, and mentoring of IRI fellows. Faculty and 
fellows have collaborated in manuscripts, and Core Fac- 
ulty are primary sponsors, mentors or consultants on 
several of the fellows' grants. Noting the emergence of 
collaborations, we launched an effort, stfll ongoing, to 
formally measure ties through an ongoing social net- 
work study. 

IRI Core faculty members have authored papers ad- 
vancing specific conceptual, substantive, or methodo- 
logical issues in implementation research. For example, 
Brownson and Proctor together with Expert Faculty 
member Colditz edited a first-of-its kind book on dis- 
semination and implementation research in health [7], 
a book for which eight other IRI faculty members con- 
tributed chapters. All Core Faculty and the IRI NIH 
program officer, David Chambers, also published a con- 
ceptual model for the study of implementation pro- 
cesses [8]. Other papers pubUshed by core faculty 
include a critical analysis of the concept of implemen- 
tation outcomes [21], a compilation of strategies for 
implementation research [22] a review of the dissemin- 
ation and implementation theories and frameworks 
[14], a review of research methods relevant to imple- 
mentation science [20], a paper on hybrid designs for 
implementation research [23], and an implementation 
conceptual framework specific to public sector mental 
health and allied health services [24]. Thus, consistent 
with the original goals, IRI faculty are advancing both 



the intellectual capital and the workforce capacity for 
the field of D&I research. 

Outcomes for the field 

The IRI has stimulated and informed other training in 
dissemination and implementation science. The IRI 
curriculum is publicly available at (http://cmhsr.wustl.edu/ 
Training/IRI/Pages/ImplementationResearchTraining.aspx). 
In 2010, a planning group developing NIH-wide training 
for dissemination and implementation research requested 
detailed information about the IRI. The IRI application 
process and Institute curriculum provided a model for the 
Training Institute on Dissemination and Implementation 
Research in Health (TIDIRH), launched in 2011 with VA 
collaboration and support. The TIDIRH is an annual, week- 
long training institute for researchers interesting in learning 
about dissemination and implementation research for a var- 
iety of health conditions and fields. TIDIRH is designed to 
encourage researchers new to D&I research to develop ap- 
plications to the trans-NIH Funding opportunity an- 
nouncements on D&I Research and related VA funding 
opportunities [25]. Each year, approximately 30 to 45 fel- 
lows participate in TIDIRH, hosted by an academic 
organization with strength in D&I research, led by a Core 
Faculty of National D&I research experts, and including 
guest faculty around key topics of interest [25]. The 
TIDIRH program is targeted to researchers at earlier stages 
of development than IRI. In fact, investigators that have re- 
ceived ROl or equivalent in implementation science are 
not encouraged to apply. Two individuals have first com- 
pleted TIDIRH, then participated in IRI. IRI faculty serve 
as core (RCB, EKP, BSM) and visiting faculty members 
(GAA) for the TIDIRH program. 

IRI Core Faculty contribute to other training initiatives 
in dissemination and implementation research: Mittman 
provides leadership for VA Enhancing Implementation 
Science (EIS) program; Aarons has helped plan and de- 
liver presentations for the NIH/OBSSR workshop on the 
use of mixed methods in implementation research, and 
Proctor and Mittman have led sessions on training at 
the NIH Meeting to Advance the Science of Dissemin- 
ation and Implementation. Finally, other impacts should 
be noted, such as faculty and feflows' contributions to 
their colleagues' work {i.e., implementation science con- 
sultation/expertise), presence and leadership of Core 
Faculty in both IRI and TIDIRH, along with local teach- 
ing and mentoring. Fellows have mentored their peers, 
and faculty experiences in IRI have shaped and strength- 
ened their own local teaching to non-IRI trainees and 
junior cofleagues in countless ways. 

Challenges and limitations 

Designing and consistently delivering high quality D&I 
training on a national level poses many chaflenges. A 
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primary challenge is that of training for a still evolving 
and rapidly developing field. To meet this challenge, Core 
Faculty have been vigilant to monitor developments in the 
field, including key publications, newly funded research 
studies, and opportunities for networking and research 
funding. The Core Faculty also work to identify gaps in the 
literature, and address them by authoring papers them- 
selves and delivering cutting edge content at each Institute. 
For example, Aarons and Glisson have presented on still-in 
-the field research on leadership for implementation and a 
multi-site trial of the ARC model, respectively. 

A second challenge is providing mentoring and support 
at a distance, since except for the two summer weeks at 
Washington University IRI fellows are based at their home 
institutions. Many local mentors are not conducting and 
may be unfamiliar with dissemination and implementation 
research. We address this challenge as follows. Core Faculty 
mentors meet regularly with their mentees via phone, sl<ype 
and email communication, and at national and local meet- 
ings and conferences. Use of these communication chan- 
nels has facilitated ongoing mentoring even while mentors 
or fellows are out of the United States. Often, the fellow 
and the local mentor participate together in the mentoring 
meetings. 

A third challenge is meeting the high demand for training. 
Each year, the pool of applicants has been extremely strong 
in terms of experience in IR and successful applications for 
federal funding— stronger than anticipated when the IRI was 
first designed. We have responded to this demand by work- 
ing actively with potential applicants, deterring many indi- 
viduals who are keenly interested by not yet prepared for 
the very competitive pool. We also refer many potential ap- 
plicants to TIDIRH, given the different selection process 
and scope of the training programs. The resultant high cali- 
ber of selected fellows contributes to the collegiality of the 
IRI, because some fellows are mid- to senior level re- 
searchers themselves. 

A fourth challenge, more pronounced in the first year, was 
securing the agreement of IR learning sites to host fellows. 
The Core Faculty found it challenging to accurately commu- 
nicate the purpose and nature of the learning visits, and 
some potential hosts declined participation because of the 
time burden and the worry that their current research pro- 
jects and processes would be exposed to outside' visitors. 
This challenge has been addressed through refinements in 
communication, as well as the nearly uniformly positive ex- 
periences of fellows and host researchers alike. 

Another challenge, indirect to IRI but very real to its 
long-term success, is preparing researchers new to im- 
plementation research within the currently challenging 
funding environment. Low pay lines make it difficult for 
scholars starting in a new field to get their grants 
funded. The pilot funds provided to fellows gives some, 
albeit very limited, support for preliminary IR studies. 



A final challenge is the heavy administrative burden of 
administering the program, given the low indirect rate 
and stringent caps on administrative and personnel ex- 
penses allowed for R25 mechanisms. The major aspects of 
IRI are the Institute, the site visits and the travel to a dis- 
semination conference. Something as basic as ensuring 
correct reimbursement of these trips entails heavy admin- 
istrative work due to the large number of fellows and fac- 
ulty and the frequency of travel. Large administrative time 
is also spent preparing applications for review and ensur- 
ing that all Core Faculty have the required materials for 
review. 

Despite the challenges, the IRI provides opportunities 
for researchers from a variety of disciplines to gain new 
perspectives on IR and the rewards brought by multidis- 
ciplinary learning experiences. Senior researchers with 
considerable experience in IR have expressed excitement 
about getting a full exposure to the range of IR expertise 
and topics brought by Expert Faculty, even some who 
are internationally based. There has been a deepening of 
collegiality and shared excitement in grappling with the 
gaps in the field and growing approaches to those gaps. 
Each cohort of fellows has developed multiple joint pub- 
lications and proposals that would not have happened 
without the intensive learning and dialogue possible in 
their IRI cohort and even cross-cohort experience. 

Conclusions and discussion 

Bold and innovative efforts are required to develop skilled 
implementation researchers. To accelerate the develop- 
ment of both human and intellectual capital, the authors 
of this paper conceived and launched the IRI. As a learn- 
ing collaborative, the IRI has coalesced senior health and 
mental health services researchers, treatment developers, 
expert methodologists, and junior researchers around 
their shared commitment to advancing their own know- 
ledge of implementation research methods (human cap- 
ital) and the field itself (intellectual capital). The IRI has 
now trained 31 new implementation researchers. Fellows 
have received two years of mentoring with the goal of de- 
veloping new implementation science research projects. 
Fellows are engaged in in-depth examinations of concep- 
tual and methodological issues, are exposed to real world 
implementation efforts and state-of-the-art implementa- 
tion research, and receive financial support for their own 
pilot studies. Based the number of implementation re- 
search grant proposals and papers produced by fellows to 
date, the IRI is proving successful in preparing new re- 
searchers who can inform the process of making evidence- 
based mental healthcare more available through real-world 
settings of care and who are advancing the field of imple- 
mentation science. 

Since the launch of the IRI, several other training pro- 
grams and infrastructure development initiatives have been 
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launched. Implementation research and training has been 
pioneered by the VA as part of the Quality Enhancement 
Research Initiative (QUERI) [26]. The VA Center for Imple- 
mentation Practice and Research Support (CIPRS), a Los 
Angeles-based resource center established in 2008, offers 
annual training in D&d science through its Enhancing Im- 
plementation Science (EIS) program, individualized consult- 
ation through a helpline and Implementation Research 
Clinic, an Implementation Research Cyber Seminar series, 
individual seminars and workshops, and various tools and 
resources to foster development and growth of implementa- 
tion research capacity. Two of the NIH supported Clinical 
and Translational Science Award (CTSA) programs offer 
specialized training and infrastructure support for D&I sci- 
ence: Washington University's CTSA program has a Dis- 
semination and Implementation Research Core (DIRC), and 
the University of California San Franciscos CTSA offers an 
Implementation and Dissemination Science (IDS) Certificate 
Program, a Master s Program in Clinical Research, and a Ph. 
D. in Epidemiology and Translational Science whose con- 
ceptual framework, training domains, courses, and compe- 
tencies are described in an article by Gonzales et al. [27]. 
The Canadian Knowledge Translation (KT) training initia- 
tive began as a KT Summer Institute in 2008, and its 
conceptualization, core competencies, and lessons learned' 
are described in several recent publications [28-31]. Straus 
et aL[31] note their inability to find national training strat- 
egies which their program could model, reflecting the need 
for a literature on implementation science training— a litera- 
ture to which the present paper contributes, as does the 
Stamatal<is et al [32] paper on insights from recent imple- 
mentation science trainees, one of whom is an IRI 'alum'. 

The final year of the current funding cycle will support 
the Core Faculty in assessing various curriculum models 
that evolve from the IRI experience, identifying new topics 
and lessons learned that will be developed to inform and 
support training experiences for trainees earlier in the 
pipeline, such as doctoral and postdoctoral students, and 
potentially for earlier levels where more specific and lim- 
ited core competencies are appropriate. It also wiU sup- 
port collection of outcomes over a longer observation 
period, as well as more nuanced analyses of outcomes and 
identification of factors associated with varying levels of 
fellows' contributions to implementation science. 
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